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Capsule  Description 


This  module  consists  of  a  comprehensive  examina¬ 
tion  of  the  technical  review  process  in  the  software 
development  and  maintenance  life  cycle.  Formal  re¬ 
view  methodologies  are  analyzed  in  detail  from  the 
perspective  of  the  review  participants,  project  man¬ 
agement  and  software  quality  assurance.  Sample  re¬ 
view  agendas  are  also  presented  for  common  types 
of  reviews.  The  objective  of  the  module  is  to  pro¬ 
vide  the  student  with  the  information  necessary  to 
plan  and  execute  highly  efficient  and  cost  effective 
technical  reviews. 


Philosophy 


This  module  provides  the  depth  required  to  fully  un¬ 
derstand  the  software  technical  review  process.  This 
process  is  a  complex  group  activity  for  which  there 
exists  an  abundance  of  basic  concepts  evolved  over 
years  of  practical  experience.  This  module  collects 
these  concepts  and  years  of  practical  experience  and 
integrates  them  from  the  software  developer,  man¬ 
ager  and  quality  assurance  perspectives. 

Many  of  the  basic  concepts  in  this  module  are  ap¬ 
plicable  to  the  introductory  course  on  Software  En¬ 
gineering  and,  thus,  can  be  used  to  support  the 
course.  In  particular,  the  rationale  for  software  tech¬ 
nical  reviews,  the  types  of  technical  reviews  which 
may  occur  on  a  project  and  an  overview  of  review 
methodologies  are  important  topics  for  this  course. 

This  module  can  also  be  used  to  support  any  other 
modules  such  as  Software  Requirements  Specifica¬ 
tion  Overview  and  Introduction  to  Design  which 
produce  documentation  that  needs  to  be  reviewed. 
Sample  checklists  and  review  agendas  are  presented 
for  many  typical  project  technical  reviews.  This 
module  also  is  important  for  those  modules  which 
describe  planning  activities  necessary  on  a  project. 


including  Software  Development  Plans  and  Software 
Quality  Assurance  Plans. 

This  module  is  also  essential  to  the  development  of 
any  Software  Testing  or  Software  Quality  Assurance 
“course”  in  which  much  of  the  contents  of  this  mod¬ 
ule  can  be  incorporated  or  assumed  to  be  prerequi¬ 
site  material. 


Objectives 


A  student  who  has  woiked  through  this  module  is 
expected  to: 

•  be  able  to  explain  the  rationale  for  soft¬ 
ware  technical  reviews; 

•  understand  the  broad  spectrum  of  review 
processes  and  their  range  from  very  for¬ 
mal  to  very  informal; 

•  describe  the  various  types  of  reviews  that 
may  take  place  on  a  project; 

•  discuss  the  role  of  internal  and  external 
standards  in  respect  to  technical  reviews; 

•  understand  the  process  for  assessing  the 
effectiveness  of  a  particular  type  of  tech¬ 
nical  review  in  a  development  process; 

•  be  cognizant  of  the  various  types  of  re¬ 
view  methodologies  which  exist  and  how 
to  select  the  appropriate  methodology  for 
a  particular  review; 

•  be  able  to  effectively  execute  the  role  of 
review  leader,  recorder,  reviewer  and 
producer, 

•  understand  techniques  for  conflict 
resolution; 

•  understand  the  planning  steps  for  effec¬ 
tive  reviews  including  how  to  select  par¬ 
ticipants,  when  to  schedule  a  review  and 
how  much  time  to  allocate  to  the  review 
process; 
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•  be  capable  of  developing  agendas  for 
various  types  of  reviews; 

•  describe  the  content  of  review  reports, 
the  various  perspectives  for  these  reports, 
the  possible  distribution  of  these  reports 
and  the  behavioral  impact  of  report  dis¬ 
tribution  for  the  producer, 

•  be  able  to  generate  action  items  as  a  con¬ 
sequence  of  a  technical  review  and  de¬ 
scribe  approaches  for  following  up  on 
action  items; 

•  appreciate  the  complex  behavioral  fac¬ 
tors  involved  in  review  processes  and  be 
able  to  utilize  techniques  designed  to 
motivate  reviewers  and  reduce  stress  of¬ 
ten  associated  with  reviews; 

•  be  capable  of  inteipreting  data  generated 
from  review  processes  in  terms  of  both 
the  software  undergoing  review  and  the 
development  process  and/or  maintenance 
process  that  produced  it. 


Prerequisite  Knowledge _ 

This  module  assumes  an  understanding  of  the  soft¬ 
ware  development  and  maintenance  process  at  a 
level  where  these  life  cycle  activities  can  be  under¬ 
stood  in  terms  of  their  reviewable  work  products. 
The  material  can,  thus,  be  integrated  into  all  relevant 
courses  including  the  Introductory  Course  on  Soft¬ 
ware  Engineering  after  the  overall  life  cycle  is  dis¬ 
cussed. 
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Module  Content 


introduction _ 

The  terms  review  process  and  software  are  used  with 
precise  meanings  in  this  section. 

A  review  process  can  be  defined  as  a  critical  evalu¬ 
ation  of  an  object.  Although  the  term  review  process 
often  has  many  connotations,  particularly  for  those 
with  industry  experience,  the  intent  of  this  module  is 
to  use  this  term  in  its  most  general  sense.  Thus, 
walkthroughs,  inspections  and  audits  can  be  viewed 
as  forms  of  review  processes. 

The  term  software  is  used  in  th»s  module  to  apply  to 
all  of  the  work  products  generated  during  the  devel¬ 
opment  and  maintenance  of  the  product  and  not  just 
the  code.  The  intent  of  this  module  is  to  describe 
review  processes  applicable  to  all  of  these  work 
products. 


Outline _ 

1.  Review  Processes  in  Society 

a.  Typical  review  processes  in  society 

b.  Roles  of  review  participants  for  typical  reviews 

c.  Training  and  preparation  of  participants  for 
typical  reviews 

d.  Reasons  why  some  reviews  are  stressful 

e.  Stress  manifestation  in  review  processes 

f.  Basic  techniques  for  minimizing  stress 

g.  Techniques  for  conflict  resolution 

h.  Factors  essential  for  successful  reviews 

2.  Rationale  for  Software  Technical  Reviews 

a.  Error  prone  software  development  and 
maintenance  process 

b.  Inability  to  test  all  software 

c.  Reviews  are  a  form  of  testing 

d.  Reviews  are  a  way  of  tracking  a  project 

e.  Reviews  provide  feedback 

f.  Educational  aspects  of  reviews 

g.  Impact  on  morale 

h.  Impact  on  maintainability 

3.  Types  of  Software  Technical  Reviews 


a.  Development  life  cycle  models 

b.  Current  standards 

4.  Behavioral  Factors 

a.  Motivating  reviewers 

b.  Small  group  theory 

c.  Minimizing  stress 

d.  Review  logistics 

5.  Formal  versus  Informal  Reviews 

a.  Terminology 

b.  Informal  reviews 

c.  Need  for  formalism 

d.  Extemal/mtemal  reviews 

e.  Responsibility  of  reviewers 

f.  Review  reporting  and  follow-up 

6.  Review  Methodologies 

a.  Walkthroughs 

b.  Inspections 

c.  Audits 

7.  Roles  of  Review  Participants 

a.  Review  leader 

b.  Recorder 

c.  Reviewer 

d.  Producer 

8.  Planning  for  the  Review  Process 

a.  Selecting  participants 

b.  Scheduling  the  review 

c.  Developing  review  agendas 

9.  Review  Reports 

a.  Management  perspective 

b.  Customer  perspective 

c.  Developer  perspective 

d.  SQA  perspective 

10.  Assessing  the  effectiveness  of  technical  reviews 

a.  Error  detection  efficiency 

b.  Cost  effectiveness 

c.  Relationship  of  reliability  assurance  techniques 

d.  Tool  support  for  review  processes 
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Annotated  Outline 


1.  Review  Processes  in  Society 

This  section  establishes  a  framework  for  viewing  soft¬ 
ware  technical  reviews  from  a  larger  scope.  The  intent 
of  this  section  is  for  the  students  to  realize  that  review 
processes  take  place  all  of  the  time  in  our  society.  Ex¬ 
amples  of  typical  reviews  include  course  final  exams, 
job  interviews,  performance  evaluations  and  IRS 
audits.  Examples  of  reviews  that  involve  evaluation  of 
team  efforts  should  also  be  presented.  In  each  type  of 
review  it  is  possible  to  identify  various  roles  that  the 
review  participants  perform.  For  each  of  these  roles 
varying  degrees  of  training  and  preparation  can  be  ob¬ 
served  depending  upon  the  type  of  review. 

This  section  also  addresses  the  notion  that  many  re¬ 
views  are  stressful  and,  therefore,  viewed  negatively  by 
the  review  participants.  Sources  of  stress  in  review 
processes,  observations  on  how  this  stress  affects  the 
review  process  as  well  as  basic  techniques  for  minimiz¬ 
ing  stress  in  reviews  should  be  discussed. 

Conflicts  are  also  a  real  part  of  many  review  processes. 
Sources  of  conflicts  in  reviews  should  be  discussed  as 
well  as  practical  techniques  for  dealing  with  these  con¬ 
flicts. 

The  section  concludes  with  a  recognition  of  the  factors 
essential  for  any  type  of  review  to  be  successful.  These 
include  careful  timing  of  the  review,  credibility  of  the 
reviewers  and  integrity  of  the  review  process. 

a.  Typical  review  processes  in  society 

b.  Roles  of  review  participants  for  typical  reviews 

c.  Training  and  preparation  of  participants  for 
typical  reviews 

d.  Reasons  why  some  reviews  are  stressful 

e.  Stress  manifestation  in  review  processes 

f.  Basic  techniques  for  minimizing  stress 

g.  Techniques  for  conflict  resolution 

h.  Factors  essential  for  successful  reviews 

2.  Rationale  for  Software  Technical  Reviews 

This  section  describes  the  importance  of  technical  re¬ 
views  in  the  software  development  and  maintenance 
life  cycle.  The  intent  of  this  section  is  for  the  students 
to  realize  that  review  processes  are  absolutely  essential 
to  the  development  and  maintenance  of  quality  soft¬ 
ware  [Weinberg84j. 

a.  Error  prone  software  development  and 
maintenance  process 

The  complexity  and  error-prone  nature  of  develop¬ 
ing  and  maintaining  software  should  be  demon¬ 
strated  with  statistics  depicting  error  frequencies  for 


intermediate  software  products.  These  statistics 
must  also  convey  the  message  that  errors  occur 
throughout  the  development  process  and  that  the 
later  these  errors  are  detected,  the  higher  the  cost  for 
their  repair  [Boehm76], 

i.  Complexity  of  software  development  and 
maintenance  processes 

ii.  Error  frequencies  for  software  work  products 

iii.  Error  distribution  throughout  development 
phases 

iv.  Increasing  costs  for  error  removal 
throughout  the  life  cycle 

b.  Inability  to  test  all  software 

This  section  must  convince  students  that  it  is  not 
possible  to  test  all  software.  Clearly  exhaustive  test¬ 
ing  of  code  is  impractical.  Current  technology  also 
does  not  exist  for  testing  a  specification  or  high  level 
design  [McKissick84].  The  idea  of  testing  a  software 
test  plan  is  also  bewildering.  Testing  also  does  not 
address  quality  issues  or  adherence  to  standards 
which  are  possible  with  review  processes. 

i.  Exhaustive  software  testing  is  impossible 

ii.  Intermediate  software  products  are  largely 
untestable 

c.  Reviews  are  a  form  of  testing 

The  section  fosters  a  recognition  by  the  students  that 
technical  reviews  are  really  a  form  of  testing  that  is 
essential  during  software  development  and  mainte¬ 
nance.  The  degree  of  formalism,  scheduling  and 
generally  positive  attitude  afforded  to  testing  must 
exist  for  software  technical  reviews  if  quality  prod¬ 
ucts  are  to  be  produced. 

i.  Objectives 

ii.  Human-based  versus  machine-based 

iii.  Attitudes  and  norms 

d.  Reviews  are  a  way  of  tracking  a  project 

The  importance  of  technical  reviews  for  tracking  a 
project  must  be  stressed.  Through  identification  of 
deliverables  with  well  defined  entry  and  exit  criteria 
and  successful  review  of  these  deliverables,  progress 
on  a  project  can  be  followed  and  managed  more  eas¬ 
ily  [Fagan76],  [McConnell84].  In  essence,  review 
processes  provide  milestones  with  teeth.  This  track¬ 
ing  is  very  beneficial  for  both  project  management 
and  customers. 

i.  Individual  developer  tracking 

ii.  Management  tracking 

iii.  Customer  tracking 

e.  Reviews  provide  feedback 
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The  instructor  should  discuss  and  provide  examples 
about  the  value  of  review  processes  for  providing 
IP  feedback  about  the  software  and  its  development 
process.  Examples  about  how  review  processes  can 
impact  the  existing  software  development  such  as  by 
identifying  weaknesses  in  the  software  that  will  re¬ 
quire  additional  validation  effort  in  the  future  must 
also  be  provided. 

i.  Product 

ii.  Process 

f.  Educational  aspects  of  reviews 

It  is  important  in  this  section  to  stress  the  educa¬ 
tional  benefits  of  performing  technical  reviews 
[McKissick84],  [McConnell84],  These  benefits  in¬ 
clude  a  better  understanding  of  the  software  by  the 
review  participants  than  can  be  obtained  by  reading 
the  documentadon  as  well  as  the  opportunity  of  ac¬ 
quiring  additional  technical  skills  by  observing  the 
works  of  others  [Hart82],  [Peele82]. 

i.  Project  understanding 

ii.  Technical  skills 

g.  Impact  on  morale 

This  section  addresses  the  impact  of  technical  re¬ 
views  on  employee  morale.  The  students  should  be 
made  aware  that  this  impact  may  be  either  positive 
or  negative.  For  some  employees,  such  as  mainte¬ 
nance  personnel,  the  reviews  may  provide  an  oppor¬ 
tunity  to  gain  visibility  of  their  work  and,  thus,  will 
be  viewed  positively.  For  others,  the  reviews  may 
be  perceived  as  an  intrusion  into  their  workplace. 

i.  Positive 

ii.  Negative 

h.  Impact  on  maintainability 

This  section  presents  the  possible  impacts  of  review 
processes  on  software  maintainability.  The  very  na¬ 
ture  of  a  review  process  requires  the  technici  as¬ 
pects  of  the  software  undergoing  review  to  be  under¬ 
standable  to  the  review  participants.  To  be  under¬ 
standable,  the  software  must  be  well  documented. 
One  of  the  acknowledged  benefits  of  technical  re¬ 
views  is  that  they  force  developers  to  produce  in¬ 
cremental  documentation  necessary  for  the  review, 
which  might  not  have  been  produced  until  the  end  of 
the  project  when  schedule  constraints  often  reduce 
the  quality  of  the  documentation  effort. 

The  review  process  itself  also  improves  the 
developer’s  general  understanding  of  the  whole  sys¬ 
tem,  which  further  facilitates  error  diagnosis  during 
maintenance  [Hart82],  Although  understandability  is 
not  the  only  attribute  of  a  maintainable  product,  the 
students  should  be  made  aware  that  review  proc¬ 
esses  at  least  contribute  in  part  to  a  more  maintain¬ 
able  product. 


i.  Better  documentation 

ii.  Standardization 

iii.  Readability 

3.  Types  of  Software  Technical  Reviews 

This  section  identifies  a  variety  of  software  technical 
reviews  that  are  possible  on  a  project  depending  upon 
the  developmental  model  followed,  the  type  of  soft¬ 
ware  product  being  produced  and  the  standards  which 
must  be  adhered  to.  Several  standards  that  affect  re¬ 
view  processes,  such  as  the  military  standards  and  the 
IEEE  standard  for  software  quality  assurance  plans, 
should  be  described  [1EEE80],  [MILS85].  Examples  of 
typical  review  processes  developed  by  organizations  to 
satisfy  these  standards  should  also  be  presented 
[McKissick84].  Other  typical  types  of  review  processes 
such  as  post  mortem  reviews,  which  are  common  to 
almost  all  projects,  should  also  be  discussed.  The  in¬ 
tent  of  this  section  is  for  the  students  to  realize  that  the 
types  of  reviews  that  must  be  performed  on  a  project 
are  dependent  upon  the  specific  intermediate 
deliverables  that  axe  produced.  For  example,  on  a  mili¬ 
tary  contract  for  an  embedded  computer  system,  certain 
review  processes  are  required  by  standards.  These, 
however,  may,  not  be  the  same  type  of  reviews  that 
must  be  performed  in  an  expert  system  development 
effort.  Examples  of  different  models  for  generating 
and  maintaining  software  should  be  provided.  An  em¬ 
phasis  on  maintenance  models  and  their  associated  re¬ 
view  processes  is  also  important  in  light  of  the  large 
percentage  of  activity  associated  with  maintenance 
functions.  Various  application  areas  should  also  be 
described  in  the  context  of  their  review  processes. 

a.  Development  life  cycle  models 

i.  Waterfall  model 

ii.  Rapid  prototyping 

iii.  Iterath  ;  enhancement 

iv.  Maintenance  activity  modeling 

b.  Current  standards 

i.  Military  standards 

ii.  IEEE  standards 

iii.  NBS  standards 

4.  Behavioral  Factors 

This  section  discusses  many  of  the  behavioral  factors 
that  must  be  dealt  with  for  a  successful  review.  The 
intent  of  this  section  is  for  the  students  to  understand 
that  any  review  process  is  a  human  activity  and  as  such 
considerable  auention  must  be  spent  on  human  inter¬ 
actions. 

a.  Motivating  reviewers 

The  first  issue  that  must  be  addressed  is  how  to  mo¬ 
tivate  reviewers.  This  is  a  complex  issue  that  ulti¬ 
mately  requires  an  organization  to  evolve  a  culture 
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in  which  review  processes  are  natural.  Included  in 
this  culture  must  be  recognition  for  good  reviewers 
and  incentives  for  performing  this  task  well.  Tech¬ 
niques  for  recognizing  good  reviewers,  such  as  peer 
evaluation,  and  possible  incentives,  such  as  cash 
awards  or  merit  increases,  should  be  described.  It  is 
also  important  to  conduct  review  processes  in  a  con¬ 
sistent  manner  in  which  every  developer  has  their 
work  reviewed.  If  any  personnel  are  immune  to  the 
review  process  for  any  reason,  serious  attitude  prob¬ 
lems  may  emerge  [Hart82]. 

i.  Review  culture 

ii.  Incentives 

iii.  Management  by  objectives 

b.  Small  group  theory 

This  section  addresses  several  key  areas  from  small 
group  theory  that  are  relevant  to  reviews.  Any 
group,  including  a  review  group,  is  subject  to  the 
problems  of  group  think,  group  deviants,  and 
domination  of  the  group  by  a  single  member.  Tech¬ 
niques  for  identifying  these  conditions  and  dealing 
with  them  in  reviews  should  be  described. 

i.  Group  deviants 

ii.  Group  think 

iii.  Dominance  by  a  review  participant 

c.  Minimizing  stress 

Techniques  fo«  minimizing  stress  for  individuals 
whose  work  is  being  reviewed  must  be  explored. 
This  also  requires  the  development  of  a  review 
culture  in  which  the  review  process  is  carefully  de¬ 
fined.  Important  issues  to  address  here  include  how 
review  information  is  utilized  and  who  has  access  to 
this  information. 

i.  Review  culture 

ii.  Management  participation  in  reviews 

iii.  Customer  participation  in  reviews 

iv.  Wcii-defined  review  processes 

d.  Review  logistics 

The  section  presents  a  description  of  review  logistics 
that  contribute  to  a  successful  review.  These  logis¬ 
tics  include  issues  regarding  timing  of  the  review, 
location  of  the  review  and  duration  of  the  review 
[Fagan76].  Many  of  these  issues  are  related  to 
limitations  of  an  individual’s  attention  span.  Other 
important  points  concern  the  number  of  reviewers 
and  their  physical  arrangement.  Much  research  has 
been  performed  suggesting  that  these  are  important 
variables. 

i.  Time  of  the  review 

ii.  Location  of  the  review 


iii.  Duration  of  the  review 

iv.  Number  of  reviewers 

v.  Physical  arrangement  of  group 

5.  Formal  versus  Informal  Reviews 

This  section  differentiates  between  formal  and  informal 
reviews.  These  terms  are  ill-defined  and  must  be 
clarified  in  this  section.  Informal  reviews  are  meant  to 
describe  the  type  of  review  that  typically  occurs  spon¬ 
taneously  among  peers  and  in  which  the  reviewers 
have  no  responsibility  and  do  not  produce  a  reveiw 
report.  Formal  reviews  are  characterized  by  carefully 
planned  meetings  in  which  reviewers  are  held  account¬ 
able  for  their  participation  in  the  review  and  in  which  a 
review  report  containing  action  items  is  generated  and 
acted  upon  [Weinberg84]. 

In  reality  there  exists  a  broad  spectrum  of  review  proc¬ 
esses  spanning  from  very  informal  peer  types  of  re¬ 
views  to  extremely  formal  and  rigorous  inspections.  In 
any  software  development  or  maintenance  project  there 
is  a  need  for  reviews  that  span  this  spectrum.  As  the 
complexity  and  size  of  a  project  increases,  more  formal 
reviews  are  necessary  for  tracking  the  project  and  ob¬ 
taining  the  right  participants  for  the  review.  This  sec¬ 
tion  also  addresses  differences  between  external  and 
internal  reviews.  External  reviews  usually  involve  the 
customer  and  are  more  formal  than  internal  reviews. 

The  intent  of  this  section  is  for  the  students  to  recog¬ 
nize  the  difference  between  formal  and  informal  re¬ 
views,  the  need  for  formal  reviews  at  critical  points  in  a 
project  and  the  ability  to  make  the  distinction  between 
what  type  of  review  is  appropriate  at  any  point  in  time. 
Global  organizational  issues  regarding  the  placement 
and  formality  of  review  processes  in  an  organization’s 
development  methodology  should  also  be  discussed. 

a.  Terminology 

b.  Informal  reviews 

c.  Need  for  formalism 

d.  Extemal/intemal  reviews 

e.  Responsibility  of  reviewers 

f.  Review  reporting  and  follow-up 

6.  Review  Methodologies 

There  are  many  variations  to  performing  technical  re¬ 
views.  Most  of  these  approaches  involve  a  group 
meeting  to  assess  a  work  product;  however,  variations 
of  reviews  exist  that  do  not  require  a  review  group 
meeting.  One  variation  is  called  a  round  robin  review 
in  which  a  work  product  is  circulated  among  reviewers 
in  a  round  robin  fashion  for  their  comments  [Hart82]. 
Most  of  this  section  focuses  on  reviews  that  involve  a 
group  meeting.  Three  major  approaches  for  perform¬ 
ing  group  oriented  technical  reviews  should  be 
presented  to  the  students. 
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It  is  important  for  the  students  to  understand  the  pri¬ 
mary  differences  among  walkthroughs,  inspections  and 
audits  as  well  as  their  respective  advantages  and  dis¬ 
advantages.  The  work  products  typically  analyzed  by 
each  of  these  processes  and  the  individuals  who  per¬ 
form  these  processes  must  also  be  discussed.  The  stu¬ 
dents  should  also  understand  how  to  select  the  appro¬ 
priate  review  methodology  for  a  particular  review. 

a.  Walkthroughs 

The  first  approach  that  should  be  described  are 
walkthroughs.  Walkthroughs  are  well-defined  by 
Yourdon  [Yourdon78],  Walkthroughs  can  be  viewed 
as  presentation  reviews  in  which  a  review  partic¬ 
ipant,  usually  the  developer  of  the  software  being 
reviewed,  narrates  a  description  of  the  software  and 
the  remainder  of  the  review  group  provides  their 
feedback  throughout  the  presentation.  These  are 
referred  to  as  presentation  reviews  because  the  bulk 
of  the  feedback  usually  only  ^  .urs  for  the  material 
actually  presented  at  the  level  it  is  presented.  Thus, 
advance  preparation  on  the  part  of  reviewers  is  often 
not  detectable  during  a  walkthrough.  Walkthroughs 
suffer  from  these  limitations  as  well  as  the  fact  that 
they  may  lead  to  disorganized  and  uncontrolled  re¬ 
views. 

Walkthroughs  may  also  be  stressful  if  the  developer 
of  the  software  is  conducting  the  walkthrough.  This 
has  lead  to  many  variations  such  as  having  someone 
other  than  the  developer  perform  the  walkthrough. 
It  is  also  possible  to  combine  multiple  reviews  into  a 
single  review  such  as  a  combined  design  and  code 
walkthrough  [Hart82]. 

i.  Presentation  reviews 

ii.  Mechanics  and  variations  of  the  process 

iii.  Benefits 

iv.  Limitations 

v.  Pitfalls 

b.  Inspections 

Inspections  should  be  presented  as  a  more  formal 
approach  that  can  be  viewed  more  as  work  product 
reviews.  Inspections  were  first  performed  by  Fagan 
at  IBM  [Fagan76].  Inspections  require  a  high  degree 
of  preparation  for  the  review  participants,  but  the 
benefits  include  a  more  systematic  review  of  the 
software  and  a  more  controlled  and  less  stressed 
meeting. 

There  are  many  variations  of  inspections,  but  all  in¬ 
clude  some  form  of  a  checklist  or  agenda  that  guides 
the  preparation  of  review  participants  and  serves  to 
organize  the  review  meeting  itself.  Inspections  are 
also  characterized  by  rigorous  entry  and  exit  require¬ 
ments  for  the  work  products  being  inspected. 

The  students  should  be  exposed  to  several  variations 


of  inspections  such  as  the  IBM  approach  [Fagan76], 
the  Bell  Labs  approach  [Ackerman83],  and  others 
[Peels82],  [Weinberg84],  [McKissick84],  The  stu¬ 
dents  should  understand  that  another  major  dif¬ 
ference  between  walkthroughs  and  inspections  is 
that  an  inspection  process  involves  the  collection  of 
data  that  can  be  used  to  feedback  on  the  quality  of 
the  development  and  review  processes  as  well  as 
influence  future  validation  techniques  on  the  soft¬ 
ware  itself.  There  are  several  references  that  pro¬ 
vide  a  good  explanation  of  the  distinction  between  a 
walkthrough  and  an  inspection  [Fagan76], 
[Freedman82],  [Quirk85]. 

i.  Work  product  reviews 

ii.  Mechanics  and  variations  of  the  process 

iii.  Benefits 

iv.  Limitations 

v.  Pitfalls 

c.  Audits 

Audits  should  also  be  described  as  an  external  type 
of  review  process.  Audits  serve  to  insure  that  the 
software  is  properly  validated  and  that  the  process  is 
producing  its  intended  results  [Walker79]. 

i.  Auditing  work  products 

ii.  Auditing  a  process 

iii.  Benefits 

iv.  Limitations 

v.  Pitfalls 

7.  Roles  of  Review  Participants 

This  section  defines  the  specific  roles  that  must  be  ex¬ 
ecuted  by  the  participants  of  a  review.  These  roles  will 
vary  depending  upon  the  specific  review  methodology 
that  is  being  followed.  These  roles  can  be  viewed  as 
being  functional,  which  implies  that  it  is  possible  in 
some  reviews  for  a  participant  to  execute  more  than 
one  role.  The  intent  of  this  section  is  for  the  students  to 
understand  the  responsibility  of  each  review  participant 
before,  during  and  after  the  review.  The  role  of  review 
participants  after  the  review  is  especially  important  to 
discuss  because  many  errors  identified  during  the  re¬ 
view  may  not  be  fixed  correctly  by  the  developer.  This 
raises  the  issue  of  who  should  follow  up  on  a  review 
and  whether  or  not  another  review  is  necessary.  Sev¬ 
eral  references  exist  for  defining  the  various  roles  of 
review  participants  [Ackerman83],  [Fagan76],  [Hart82], 
[Remus83],  [Peele82],  [Weinberg84],  [McKissick84]. 

a.  Review  leader 

The  review  leader  is  the  one  individual  responsible 
for  the  review.  This  role  requires  scheduling  the 
review,  conducting  an  orderly  review  meeting  and 
preparing  the  review  report.  The  review  leader  may 
also  be  responsible  for  ensuring  that  action  items  are 
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properly  handled  after  the  review  process.  Review 
leaders  must  possess  both  technical  and  interper¬ 
sonal  management  characteristics.  The  interpersonal 
management  qualities  include  leadership  ability, 
mediator  skills  and  organizational  talents.  The  re¬ 
view  leader  must  keep  the  review  group  focused  at 
all  times  and  prevent  the  meeting  from  becoming  a 
problem  solving  session. 

i.  Responsibilities 

ii.  Leadership  /  moderation  skills 

iii.  Preparation 

iv.  Guidelines 

b.  Recorder 

The  recorder  role  in  the  review  process  guarantees 
that  all  information  necessary  for  an  accurate  review 
report  is  preserved.  The  recorder  must  digest  com¬ 
plicated  discussions  and  capture  their  essence  in  ac¬ 
tion  items.  The  students  should  be  lead  to  under¬ 
stand  that  the  role  of  recorder  is  clearly  a  technical 
function  and  one  that  cannot  be  performed  by  a 
secretary.  This  section  should  also  discuss  tech¬ 
niques  for  insuring  the  reliability  of  the  information 
gathered  by  the  recorder  such  as  taping  the  review, 
having  two  recorders  and  reviewing  the  review 
minutes. 

i.  Responsibilities 

ii.  Preparation 

iii.  Guidelines 

c.  Reviewer 

The  reviewer  role  is  to  objectively  analyze  the  soft¬ 
ware  and  to  be  accountable  for  the  review.  In  an 
inspection  methodology,  the  reviewer  must  spend 
considerable  time  preparing  for  the  review.  Guide¬ 
lines  for  reviewers,  such  as  the  importance  for  a 
reviewer  of  keeping  in  mind  that  the  software  is 
being  reviewed  and  not  the  producer  of  the  software, 
should  also  be  presented.  Techniques  for  reviewers 
to  pose  their  questions  in  constructive  ways  should 
also  be  addressed. 

i.  Responsibilities 

ii.  Preparation 

iii.  Guidelines  for  reviewers 

d.  Producer 

The  producer  role  varies  depending  upon  the  review 
methodology.  In  a  walkthrough,  the  reviewer  may 
actually  lead  the  meeting  in  an  organized  discussion 
of  the  software.  A  high  degree  of  preparation  and 
planning  is  needed  in  a  walkthrough  to  present  mate¬ 
rial  at  the  proper  level  and  pace.  In  an  inspection 
methodology,  the  producer  must  also  be  highly  pre¬ 
pared  to  respond  to  all  points  brought  up  by  the  re¬ 
view  team.  The  attitude  of  the  producers  is  also 


important;  it  is  essential  that  they  do  not  take  a 
defensive  approach. 

i.  Responsibilities 

ii.  Preparation 

iii.  Guidelines 

8.  Planning  for  the  Review  Process 

This  section  details  the  planning  necessary  for  a  suc¬ 
cessful  review.  This  planning  can  be  described  at  both 
the  organizational  level  and  the  specific  review  level. 
At  the  organizational  level,  the  students  should  be  ad¬ 
vised  of  the  planning  necessary  by  management  to 
identify  the  appropriate  number  and  types  of  reviews 
that  are  to  be  performed  for  the  project  Resources 
must  also  be  allocated  for  accomplishing  these  reviews. 
These  resources  are  normally  allocated  during  the  crea¬ 
tion  of  the  Software  Development  or  Software  Quality 
Assurance  Plans. 

a.  Selecting  participants 

At  the  specific  review  level,  the  students  should  be 
presented  with  the  detailed  planning  issues  that  must 
be  addressed  for  a  successful  review.  This  planning 
entails  selecting  participants  and  their  respective 
roles,  scheduling  the  review  and  developing  a  re¬ 
view  agenda.  There  are  many  issues  involved  in  the 
selection  of  review  participants.  Selecting  partic¬ 
ipants  is  a  complex  task  that  normally  is  performed 
by  management  with  technical  input  When  select¬ 
ing  review  participants  care  must  be  exercised  to 
insure  that  each  aspect  of  the  software  under  review 
can  be  addressed  by  at  least  some  subset  of  the  re¬ 
view  team. 

The  students  should  also  be  made  aware  of  problems 
that  may  develop  if  a  review  group  becomes  too 
large  to  hold  a  reasonable  meeting.  In  order  to  mini¬ 
mize  stress  and  possible  conflicts  in  review  proc¬ 
esses,  it  is  important  to  discuss  the  role  that  a  pos¬ 
sible  reviewer  plays  in  the  organization.  This  role 
may  be  either  a  formal  recognized  role,  such  as  man¬ 
ager,  or  an  informal  role,  such  as  “spy”  for  manage¬ 
ment. 

i.  Responsibility  for  review  participant 
selection 

ii.  Review  group  size 

iii.  Technical  expertise 

iv.  Formal  vs.  informal  status  of  reviewers  in 
the  organization 

b.  Scheduling  the  review 

Scheduling  issues  regarding  the  timing  of  a  review 
must  also  be  presented  to  the  students.  These  in¬ 
clude  the  fact  that  scheduling  a  review  should 
ideally  take  place  soon  after  a  producer  has  com¬ 
pleted  the  software  but  before  additional  effort  is 
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expended  on  work  dependent  upon  the  software. 

The  problems  of  allocating  sufficient  time  to  a  re¬ 
view  process  should  also  be  explored.  One  of  these 
problems  stems  from  the  difficulty  in  estimating  the 
time  needed  to  perform  the  review.  This  problem  is 
analogous  to  that  of  estimating  the  time  that  any 
meeting  will  last.  The  approach  that  must  be  taken 
is  the  same  as  that  for  estimating  the  time  to  be 
allocated  for  any  meeting;  that  is,  an  agenda  must  be 
formulated  and  time  estimated  for  each  agenda  item. 

Another  problem  to  address  concerning  scheduling 
involves  the  duration  of  the  review  and  problems 
that  may  occur  if  the  review  is  too  long.  This  re¬ 
quires  review  processes  to  be  focused  in  terms  of 
their  objectives.  Review  participants  must  under¬ 
stand  these  review  objectives  and  their  implications 
in  terms  of  actual  review  time  as  well  as  preparation 
time  before  committing  to  the  review. 

The  students  should  also  be  made  aware  that  some 
organizations  have  collected  defect  data  information 
that  suggests  guidelines  for  allocating  time  for  a  re¬ 
view  process.  This  data  often  takes  the  form  of  code 
inspection  rates  [Buck83].  Others  have  collected 
data  that  suggests  a  certain  percentage  of  develop¬ 
ment  effort  is  allocated  to  the  review  process 
[McConnell84], 

i.  Management  commitment 

ii.  Ideal  review  time 

iii.  Allocating  sufficient  time  for  the  review 
process 

c.  Developing  review  agendas 

Another  objective  of  this  section  is  for  the  students 
to  understand  the  importance  of  developing  review 
agendas  and  to  be  able  to  create  and  refine  an 
agenda  for  a  particular  review.  The  development  of 
a  review  agenda  must  be  done  prior  to  the  review  by 
the  review  leader  and  the  producer.  Although  re¬ 
view  agendas  are  specific  to  any  particular  product 
and  the  objective  of  its  review,  generic  agendas  can 
and  should  be  produced  for  related  types  of  prod¬ 
ucts.  These  agendas  may  take  the  form  of  check¬ 
lists.  For  example,  generic  code  agendas  and  inter¬ 
face  specification  agendas  can  be  developed.  These 
generic  agendas  can  become  standardized  as  the  for¬ 
mat  and  content  of  software  development  work 
products  mature. 

The  support  materials  for  this  module  will  contain 
sample  agendas  for  many  types  of  reviews. 

i.  Refining  the  scope  of  the  review 

ii.  Checklists 
9.  Review  Reports 

This  section  describes  the  contents  of  a  review  report 


The  intent  of  this  section  is  for  the  students  to  under¬ 
stand  the  type  of  information  that  is  necessary  to  cap¬ 
ture  from  a  review  a  report  The  format  of  reports  is 
not  important  although,  numerous  examples  of  reports 
should  be  provided  and  will  be  included  in  the  appen¬ 
dix.  References  containing  sample  reports  include 
[Ackerman83,  Buck83,  Fagan76,  Weinberg84).  These 
contents  can  best  be  discussed  by  examining  them  from 
a  management  perspective,  a  customer  perspective,  a 
developer  perspective  and  a  SQA  perspective. 

a.  Management  perspective 

From  a  management  perspective,  the  review  report 
serves  as  a  summary  of  the  review  that  highlights 
what  was  reviewed,  who  did  the  reviewing  and  what 
was  their  assessment  It  is  somewhat  controversial 
as  to  whether  or  not  management  should  be  con¬ 
cerned  with  actual  action  items.  Clearly,  manage¬ 
ment  does  need  an  estimate  of  when  all  action  items 
will  be  resolved  to  successfully  track  the  project 

i.  Technical  review  summaries 

ii.  Project  tracking 

b.  Customer  perspective 

The  customer  may  be  interested  in  analyzing  review 
reports  for  some  of  the  same  reasons  as  the  manager 
(i.e.,  for  tracking  the  project).  The  customer  will 
also  often  be  concerned  with  examining  the  quality 
of  intermediate  work  products  in  an  effort  to  monitor 
the  development  organization’s  progress. 

c.  Developer  perspective 

In  the  analysis  of  a  review  report  from  a  developer 
perspective,  the  critical  information  is  contained  in 
the  action  items.  These  items  may  correspond  to 
actual  errors,  possible  problems,  inconsistencies  or 
other  considerations  that  the  developer  must  address. 

i.  Action  items 

d.  SQA  perspective 

The  SQA  perspective  of  a  review  report  is  twofold. 
First,  SQA  must  ensure  that  all  action  items  on  the 
review  report  are  addressed.  SQA  should  also  be 
concerned  with  analyzing  the  data  on  the  review 
forms  and  classifying  defects  to  improve  the  soft¬ 
ware  development  and  review  processes.  Many  pos¬ 
sible  classifications  of  defects  exist  and  examples 
should  be  cited  [Ackerman83].  There  is  also  a 
variety  of  interpretations  that  can  be  made  from  ac¬ 
curate  review  data  reporting  that  must  be  discussed 
[Buck83].  For  example,  a  high  number  of  specifi¬ 
cation  errors  may  suggest  a  lack  of  rigor  or  time  in 
the  requirements  specifications  phase  of  the  project 
Information  regarding  the  type  of  review,  its  partic¬ 
ipants,  its  agenda  and  its  scheduling  should  also  be 
recorded  in  order  to  facilitate  improved  review  plan¬ 
ning  activities.  In  essence,  the  information  on  re- 
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view  reports  should  be  utilized  to  evaluate  both  the 
software  and  its  development  process.  This  is  most 
often  performed  during  some  sort  of  post  mortem 
review  of  a  project. 

i.  Data  collection 

ii.  Action  item  follow-up 

10.  Assessing  the  effectiveness  of  technical  reviews 

This  section  presents  approaches  for  evaluating  the  ef¬ 
fectiveness  of  technical  reviews  as  well  as  actual  re¬ 
view  data  collected  by  organizations  [Remus83], 
[McKissick84],  Prior  to  this  section,  the  student  should 
have  beat  made  aware  of  the  relative  merits  of  review 
processes  and  the  mechanics  for  executing  them.  The 
objective  of  this  section  is  to  provide  the  students  with 
the  knowledge  of  how  to  actually  evaluate  particular 
review  processes  in  an  organization.  One  informal 
method  of  assessing  the  effectiveness  of  review  proc¬ 
esses  is  to  observe  their  impact  on  the  testing  process. 
Some  studies  report  that  effective  reviews  reduce  total 
test  time  and  production  failures  [Peele82],  Other 
studies  have  attempted  to  evaluate  the  effectiveness  of 
reviews  using  statistical  techniques.  This  effectiveness 
figure  is  then  utilized  to  predict  the  number  of  remain¬ 
ing  defects  in  the  software  [Remus79].  Two  quanti¬ 
tative  metrics  for  accomplishing  this  task,  error  detec¬ 
tion  efficiency  and  error  detection  cost  effectiveness, 
should  also  be  presented. 

a.  Error  detection  efficiency 

Error  detection  efficiency  is  a  simple  metric,  which 
examines  the  ratio  of  the  defects  detected  by  a  re¬ 
view  process  to  the  number  of  defects  that  were 
detectable.  This  metric  should  be  applied  over  a 
large  number  of  reviews  in  a  statistical  manner.  The 
determination  of  the  number  of  defects  detectable  by 
the  review  process  can  only  accurately  be  calculated 
after-the-fact.  Techniques  for  estimating  this  num¬ 
ber  should,  however,  be  presented.  These  tech¬ 
niques  include  industry  averages,  reliability  meas¬ 
ures  and  error-seeding  techniques. 

i.  Defect  data  collection 

ii.  Error  classification  schemes 

b.  Cost  effectiveness 

Error  detection  cost  effectiveness  is  a  more  complex 
metric,  which  examines  the  ratio  of  the  costs  saved 
by  the  review  process  to  the  actual  cost  of  the  proc¬ 
ess.  The  actual  cost  of  a  review  process  can  be 
measured  several  ways,  typically  using  some  varia¬ 
tion  of  manhours.  The  costs  saved  by  a  review  is 
much  harder  to  quantify  and  usually  requires  a 
statistical  analysis  of  the  errors  detected  by  the  re¬ 
view,  where  these  errors  may  have  been  detected  if 
the  review  was  not  held  and  the  cost  associated  with 
fixing  the  error  at  a  later  stage  than  if  it  was  caught 
by  the  review.  A  key  objective  of  this  section  is  for 


the  students  to  understand  these  types  of  metrics  and 
recognize  that  particular  review  processes  in  an  or¬ 
ganization  may  not  be  efficient  or  cost  effective. 

i.  Measuring  costs  saved  by  the  review 

ii.  Calculating  the  actual  cost  of  a  review 

c.  Relationship  of  reliability  assurance  techniques 

The  relationship  of  various  types  of  reliability  as¬ 
surance  techniques,  which  include  both  testing  and 
reviews,  must  also  be  discussed.  Examples  and  data 
indicating  the  relative  effectiveness  of  testing  versus 
review  processes  at  particular  points  in  a  project 
should  be  presented  [Remus79],  [Remus83].  In  par¬ 
ticular,  the  impact  of  review  and/or  testing  processes 
at  one  point  in  a  project  with  reviews  and/or  testing 
processes  at  later  points  in  the  project  must  be  ex¬ 
amined.  Reliability  assurance  techniques  must  be 
coordinated  to  maximize  effectiveness. 

i.  Reviews 

ii.  Testing 

d.  Tool  support  for  review  processes 

The  role  of  tools  in  terms  of  improving  the  effec¬ 
tiveness  of  review  processes  should  also  be  dis¬ 
cussed.  As  tools  become  available  to  perform  some 
of  the  tasks  previously  performed  by  humans,  the 
cost  effectiveness  of  review  processes  increases. 
Examples  of  how  this  is  occuring  should  be  cited.  A 
simple  example  is  the  utilization  of  a  compiler  to 
detect  syntax  errors  in  code,  thus  alleviating  this  task 
for  the  reviewers.  Design  and  specification  consis¬ 
tency  checkers  are  another  example  of  tool  support 
for  review  processes. 
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Teaching  Considerations 


Recommended  Support  Materials 

To  support  the  instruction  of  this  module,  example 
review  report  forms,  suggested  review  participants 
lists,  sample  review  agendas,  applicable  checklists, 
and  detailed  information  about  various  types  of  re¬ 
views  should  be  provided.  Such  materials  will  even¬ 
tually  be  available  in  the  support  materials  package 
for  this  module. 

A  video  tape  lasting  approximately  50  minutes  that 
addresses  the  behavioral  factors  in  review  processes 
would  be  valuable.  This  tape  would  consist  of  nar¬ 
rated  segments  of  reviews  that  illustrate  both  good 
and  bad  behavioral  characteristics.  The  purpose  of 
the  video  tape  is  to  explain  some  of  the  behavioral 
factors  in  review  processes  in  a  more  professional 
manner  than  a  typical  instructor  untrained  in  psy¬ 
chology  or  sociology. 


Exercises 


Several  kinds  of  exercises  have  been  found  to  be 
useful  and  effective  in  teaching  this  material.  These 
are  described  under  the  appropriate  heading  from  the 
outline  presented  previously. 

Review  Processes  in  Society.  Have  the  student 
think  about  typical  non-technical  reviews  encoun¬ 
tered  every  day  in  our  society  and  address  each  of 
the  applicable  subtopics  outlined  in  this  section  per¬ 
tinent  to  these  reviews. 

Types  of  Software  Technical  Reviews.  Have  the 
student  examine  several  different  types  of  products, 
such  as  an  embedded  system  and  an  expert  system. 
For  each  of  these  products,  identify  appropriate  re¬ 
views  that  are  applicable,  assuming  the  project  is 
being  developed  with  a  particular  life  cycle  model. 

Roles  of  Review  Participants.  The  exercises  in 
this  section  involve  actual  reviews.  Each  student 
should  have  the  opportunity  of  playing  each  of  the 
review  participant  roles  for  both  an  inspection  and  a 
walkthrough.  The  students  should  also  have  the  op¬ 
portunity  of  observing  reviews  and  reporting  back 
problems  that  they  thought  hindered  these  reviews  as 
well  as  actions  that  enhanced  the  reviews. 


Planning  for  the  Review  Process.  Various  ex¬ 
ercises  can  be  designed  to  provide  the  students  with 
a  better  understanding  of  selecting  review  partici¬ 
pants,  estimating  the  time  for  the  review  and  devel¬ 
oping  a  workable  review  agenda.  A  reasonable  ex¬ 
ercise  would  be  to  examine  the  Elevator  Control 
Problem,  which  is  one  of  the  continuing  examples 
under  development  at  the  SEI,  and  define  several  re¬ 
views  such  as  the  specification  review,  the  high  level 
design  review  and  a  test  plan  review.  For  each  of 
these  reviews,  participants  should  be  identified,  a 
workable  agenda  established  and  a  time  estimate  for 
the  review  derived. 

Review  Reports.  A  reasonable  set  of  exercises  for 
this  section  would  be  to  present  the  students  with 
comprehensive  summary  data  collected  from  various 
review  processes  and  ask  them  to  interpret  the  data 
in  terms  of  both  the  software  and  its  development 
process.  Another  variation  of  this  same  scheme 
would  be  to  ask  the  students  how  they  would  assess 
any  weaknesses  in  an  organization’s  development 
approach  and  evaluate  the  impact  of  changes  tar¬ 
geted  to  improve  these  weaknesses. 
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described. 

Remus83 

Remus,  H.  “Integrated  Software  Validation  in  the 
View  of  Inspections/Reviews.”  Software  Valida¬ 
tion,  Inspection-Testing-Verification-Alternatives: 
Proceedings  of  the  Symposium  on  Software 
Validation.  Amsterdam:  North-Holland,  Sept.  1983, 
57-63. 

Abstract:  The  software  development  process  is 
looked  at  as  to  the  specific  contribution  of 
inspections/reviews  to  the  discovery  of  wrong  de¬ 
sign  directions  or  implementations.  The  benefits  are 
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chanics  of  the  process  are  presented.  Psychological 
issues  for  walkthroughs  are  also  noted. 
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education.  A  curriculum  modulo  (CM)  identifies  and  outlines  the  content  of  a  specific  topic  area,  and  is  intended  to  be 
used  by  an  instructor  in  designing  a  course.  A  support  materials  package  (SM)  contains  materials  related  to  a  module 
that  may  be  helpful  in  teaching  a  course.  An  educational  materials  package  (EM)  contains  other  materials  not  necessarily 
related  to  a  curriculum  module.  Other  publications  include  software  engineering  curriculum  recommendations  and  oourse 
designs. 

SEI  educational  materials  are  being  made  available  to  educators  throughout  the  academic,  industrial,  and  government 
communities.  The  use  of  these  materials  in  a  course  does  not  in  any  way  constitute  an  endorsement  of  the  course  by  the 
SEI,  by  Carnegie  Mellon  University,  or  by  the  United  States  government 

Permission  to  make  copies  or  derivative  works  of  SEI  curriculum  modules,  support  materials,  and  educational  materials  is 
granted,  without  fee,  provided  that  the  copies  and  derivative  works  are  not  made  or  distributed  for  direct  commercial 
advantage,  and  that  al  copies  and  derivative  works  cite  the  original  document  by  name,  author's  name,  and  document 
number  and  give  notice  that  the  oopying  is  by  permission  of  Carnegie  Mellon  University. 

Comments  on  SEI  educational  materials  and  requests  for  additional  information  should  be  addressed  to  the  Software 
Engineering  Curriculum  Project,  Software  Engineering  Institute,  Carnegie  Mellon  University,  Pittsburgh,  Pennsylvania 
15213.  Electronic  mail  can  be  sent  to  education@sei.cmu.edu  on  the  Internet. 


Curriculum  Modules  (*  Support  Materials  available) 

CM-1  (superseded  by  CM-19) 

CM-2  Introduction  to  Software  Design 
CM-3  The  Software  Technical  Review  Process* 

CM-4  Software  Configuration  Management* 

CM- 5  Information  Prolection 
CM-S  Software  Safety 
CM-7  Assurance  of  Software  Quatity 
CM-ti  Formal  Specification  of  Software* 

CM-9  Unit  Testing  and  Analysis 

CM-10  Models  of  Software  Evolution:  Life  Cycle  and  Process 
CM-1 1  Software  Specifications:  A  Framework 
CM- 12  Software  Metrics 

CM- 13  Inkoduction  to  Software  Verification  and  Validation 
CM- 14  Intelectua!  Property  Protection  for  Software 
CM- 15  Software  Development  and  Licensing  Contracts 
CM-16  Software  Development  Using  VDM 
CM-17  User  Interlace  Development* 

CM-18  (superseded  by  CM-23] 

CM-19  Software  Requirements 
CM- 20  Formal  Verification  of  Programs 
CM-21  Software  Project  Management 
CM-22  Software  Design  Methods  for  Real-Time  Systems* 
CM-23  Technical  Writing  for  Software  Engineers 
CM- 24  Concepts  of  Concurrent  Programming 
CM- 25  Language  and  System  Support  for  Concurrent 
Programming* 

CM-26  Understanding  Program  Dependencies 


Educational  Materials 

EM-1  Software  Maintenance  Exercises  for  a  Software 
Engineering  Project  Course 

EM-2  APSE  Interactive  Monitor:  An  Artifact  for  Software 
Engineering  Education 

EM-3  Reading  Computer  Programs:  Instructor's  Guide  and 
Exercises 


